Data management in a data storage system

ABSTRACT

The present disclosure provides a method in a data storage system. The method includes defining a plurality of jobs for a command received from a host. Each of the plurality of jobs is associated with one or more of a plurality of data storage resources of the data storage system. The plurality of jobs have a defined order that is a function of addresses of data in the plurality of data storage resources. The method also includes issuing the plurality of jobs to the associated data storage resources and receiving information from the data storage resources for the plurality of jobs. The information is received by a controller of the data storage system for the jobs in an order that is different than the defined order. The method includes transmitting the received information to the host for the plurality of jobs in the defined order.

BACKGROUND

A data storage system is an example of a system having data resources.For example, a data storage system includes one or more devices havingat least one medium for data storage. The data storage system caninclude one or more types of storage mediums such as, but not limitedto, hard discs, floppy discs, magnetic discs, optical discs, magnetictapes, solid-state storage components, and/or combinations thereof. Forinstance, an exemplary data storage system can comprise a hard discdrive (HDD), a solid-state drive (SDD), a “hybrid” drive (e.g., a hybridhard drive (HHD)), to name a few.

In one example, the data storage system includes a controller that isconfigured to receive data and commands from a host and implement dataoperations to the storage media in the data storage system based on thecommands. The data storage system can include a plurality of devices andcomponents having memory accessible by the controller. For instance, asolid-state drive (SDD) can include a plurality of data memory devices,such as flash memory chips, having solid-state memory accessible by acontroller of the solid-state drive (SDD).

The discussion above is merely provided for general backgroundinformation and is not intended to be used as an aid in determining thescope of the claimed subject matter.

SUMMARY

In one exemplary embodiment, a method in a data storage system isprovided. The method includes defining a plurality of jobs for a commandreceived from a host. Each of the plurality of jobs is associated withone or more of a plurality of data storage resources of the data storagesystem. The plurality of jobs have a defined order that is a function ofaddresses of data in the plurality of data storage resources. The methodincludes issuing the plurality of jobs to the associated data storageresources and receiving information from the data storage resources forthe plurality of jobs. The information is received by a controller ofthe data storage system for the plurality of jobs in an order that isdifferent than the defined order. The method includes transmitting thereceived information to the host for the plurality of jobs in thedefined order.

In one exemplary embodiment, a data storage system is provided. The datastorage system includes an interface for receiving a command for a dataoperation, a plurality of data memory devices, and a controller. Thecontroller is configured to generate a plurality of jobs for thecommand, issue the plurality of jobs to the data memory devices, andreceive information from the data memory devices for the plurality ofjobs. Each of the plurality of jobs are associated with one or more ofthe data memory devices. The plurality of jobs have a defined order thatis a function of addresses of data in the data memory devices. Theinformation is received at the controller from the data memory devicesfor the jobs in an order that is different than the defined order.

In one exemplary embodiment, a data storage device is provided. The datastorage device includes a plurality of data storage resources and acontroller configured to identify at least one command for a dataoperation and generate a plurality of jobs for the data operation basedon the at least one command and the plurality of data storage resources.The controller is configured to issue the plurality of jobs to theplurality of data storage components and includes a data managementcomponent configured to concurrently manage completion informationpertaining to completion of the plurality of jobs.

These and various other features and advantages will be apparent from areading of the following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating an exemplary data storagesystem.

FIG. 2 is a schematic diagram illustrating one embodiment of the datastorage system shown in FIG. 1.

FIG. 3 is a diagram of a data structure representing an exemplary datacommand, under one embodiment.

FIGS. 4A and 4B is a flow diagram illustrating a method for concurrentdata management using the exemplary data structure illustrated in FIG.3.

FIGS. 5A and 5B is a flow diagram illustrating a method for concurrentdata management for multiple data commands.

FIG. 6 is a diagram of a data structure representing an exemplary datacommand, under one embodiment.

FIGS. 7A and 7B is a flow diagram illustrating a method for concurrentdata management using the exemplary data structure illustrated in FIG.6.

DETAILED DESCRIPTION

Aspects of the present disclosure relate generally to systems havingconcurrent data resources and more specifically, but not by limitation,to concurrent data management in a data storage system.

One aspect of data management in a data storage system, for example,generally relates to management of data flows from/to data resources,such as data storage media. For instance, data management can comprisetracking completion of data write operations to one or more data storagemedia. In another instance, for data read operations data management cancomprise tracking data retrieved from one or more data storage resourcesand transferring the data to a host.

Data commands received from a host system or device, for example, caninclude constraints related to completion of the associated dataoperation(s). For example, for a read command, communication of readdata from the data storage system to the host can require that the readdata is transmitted to the host in an order and manner that is expectedby and acceptable to the host. For instance, the order of data can bebased on a function of logical block addresses of the data. In suchcases, data management can comprise ensuring that the data is retrievedfrom the data storage devices and “throttled” to the host appropriately.The terms “throttled” or “throttling” refer to a process of ensuringthat the data is retrieved from the data storage devices and transferredto the host in an order acceptable to the host. For example, in oneembodiment data retrieved from the data storage devices is onlytransferred to the host when each respective part of the command iscompletely ready to be transferred to the host. The host may requirethat the data is received from the data storage system in an order basedon the logical block addresses of the retrieved data.

In some data storage systems, data management is essentially“single-threaded” in that there is a single data resource (e.g., aread/write head) that can deliver data to/from a controller, buffer,cache, host interface, etc., at a time. On the other hand, some datastorage systems include a plurality of data resources that are“concurrent” in that multiple data resources can be instructed to carryout data operations (or portions thereof) at substantially the same timeand/or during overlapping time intervals. For instance, in a datastorage system comprising an exemplary solid-state drive there can betens to hundreds (or more) of concurrent data resources. For example,the solid-state drive can include a plurality of independent solid-statedata memory devices (e.g., flash chips). Data commands (or portionsthereof) can be provided from a drive controller to a number of the datamemory devices concurrently. To illustrate, in one example thesolid-state drive employs a mapping of host logical block addresses(LBAs) to physical locations. A set of LBAs for a single read command,for example, may be mapped to locations distributed across multiple datamemory devices allowing the multiple data memory devices to service thesingle read command. Likewise, LBAs for multiple received read commandscan also be mapped to locations distributed across the multiple datamemory devices.

As mentioned above, the one or more read commands can include associatedconstraints (i.e., a host-expected data order) related to gathering ofthe read data from the data memory devices. In such cases, datamanagement can comprise ensuring that the retrieved data is throttled tothe host appropriately (i.e., held up until it can be delivered in thehost-expected data order). However, retrieving the data from the datamemory devices in-order can significantly limit performance of the datastorage system.

In accordance with one aspect described herein, data management in adata storage system having a plurality of concurrent resources isprovided. As used herein, the management of data across “concurrent”resources is referred to as concurrent data management. In oneembodiment, concurrent data management enables data to be retrieved fromthe plurality of concurrent resources for one or more read operations,for example, in an order that is different (i.e., “out-of-order”) than adata order defined for the read operation(s).

FIG. 1 is a schematic diagram of an exemplary data computing system 100including a data storage system 108 having a plurality of concurrentdata resources 112. As illustrated, a host system 101 includes aprocessor 102 connected to a system bus 103 which also can be connectedto input/output (I/O) devices 104, such as a keyboard, monitor, modem,storage device, or pointing device. The system bus 103 is also coupledto a memory 106, which can include a random access volatile memory, suchas dynamic random access memory (DRAM). The system bus 103 is alsocoupled to the data storage system 108 for communicating data and/orcommands between data storage system 108 and host system 101.

Data storage system 108 includes a controller 110, which can be coupledto the processor 102 via a connection through the system bus 103. It isnoted that in some systems this connection is made through one or moreintermediary devices, such as a host bus adapter or a bridge.

Controller 110 communicates with the plurality of data resources 112over one or more channels (i.e., buses) 114. In the illustratedembodiment, the data resources 112 comprise a plurality of data storagedevices (illustratively, solid-state data memory devices). In oneexample, data storage system 108 comprises a solid-state drive anddevices 112 comprise semi-conductor based devices, such as flash memory.In other embodiments, the data storage system 108 can also includevolatile and/or non-solid-state memory. For example, data storage system108 can comprise a disc drive and/or a “hybrid” drive includingsolid-state components and hard disc components.

FIG. 2 is a schematic diagram illustrating one embodiment of datastorage system 108. Data storage system 108 includes controller 110 thatis configured to store information to and retrieve information from theplurality of solid-state data memory devices (illustratively flashmemory) 112.

In one embodiment, each of the devices 112 comprise an independent flashagent that is able to perform a data operation, or portion thereof,associated with a command received by the controller. For example, eachflash agent 112 is configured to perform all, or a portion of, a dataread, a data write operation, etc. Further, the data operation does nothave to include a data transfer. For example, the data operation caninclude a data erase operation, such as an erase operation on a flashchip.

In one embodiment, each device 112 is identified by an assigned logicalunit number (LUN). Each flash agent 112 can comprise one or more flashchips, for example. Alternatively, or in addition, one or more flashagents 112 can be provided on the same flash chip. In this manner,multiple logical storage units can be provided within a single die orpackage, for example. For instance, each flash agent 112 can include aseparate flash chip comprising a semiconductor package having one ormore semiconductor dice provided in a housing, for example.

Each device 112 can include an interface for communicating informationwith memory interface 220, control circuitry, and a storage area (havinga particular capacity based on the design of the device components). Forexample, in one embodiment the storage area of one or more flashagent(s) 112 is capable of storing 1 mebibyte (MiB). In anotherembodiment, one or more flash agent(s) 112 are configured to store morethan or less than 1 MiB.

However, it is noted that data memory devices 112 can have any suitablephysical structure. For instance, each of data memory devices 112 can beprovided on the same semiconductor die (e.g., the same piece ofsilicon). In another instance, one or more of data memory devices 112are provided on different semiconductor die (e.g., different pieces ofsilicon). Further, it is noted that data storage system 108 can includeany number of data memory devices 112. For example, in one embodimentdata storage system 108 includes 4 to 256 data memory devices 112.However, less than 4 or more than 256 data memory devices 112 can beutilized.

Controller 110 includes memory interface 220 (illustratively a flashmemory interface) that is coupled to the data memory devices 112 via oneor more channels (i.e., busses) 114 for communicating commands and/ordata. In one embodiment, channels 114 comprise 1 to 24 flash channels.However, any number of channels and/or connection topologies can beutilized. Channels 114 can comprise data busses, address busses, and/orchip select busses, for example.

While FIG. 2 illustrates a particular channel configuration, it is notedthat the attachment methodology between the controller 110 and datamemory devices 112 can be of any suitable form.

The controller 110 is communicatively coupled to a host, such as hostsystem 101 illustrated in FIG. 1, via a host interface 216 that canreceive and send commands, status information, and data to the host. Thehost interface 216 can pass commands to a control circuit 230 ofcontroller 110 for processing and also store received data in a buffermemory 232. The buffer memory 232 provides the received data to thememory interface 220.

The memory interface 220 can receive data from the buffer memory 232 tobe written to one or more of the data memory devices 112 and receiveaddress bits from the control circuit 230. The memory interface 220 canassert corresponding data and address bits with appropriate timing andformat to a selected data memory device 112. Memory interface 220 canalso receive previously stored data from any storage locations (e.g.,pages, blocks, etc.) of data memory devices 112.

In the flash memory example of FIG. 2, when a logical sector of data(i.e., a portion of data from host system 101 having an associatedlogical block address (LBA)) is to be written to memory 112, flashcontroller 110 identifies the physical address of the physical block towhich the data will be written. The logical block address (LBA) is theaddress that the host system uses to read or write a block of data todata storage system 108. The physical block address is the fixed,physical address of a block in the storage component. In one example,the controller 110 can store a mapping of the logical addresses to thecorresponding physical addresses in a translation/mapping component 238.The mapping information is utilized for data access operations to locaterequested data in the memory devices 112. In another example, component238 can operate as a cache for the mapping information. For instance,the mapping information can be stored to or otherwise associated withdata memory devices 112. In this manner, the mapping information can befetched from (on a read) or updated to (on a write) the solid-state datamemory device(s) associated with the command.

Data memory devices 112 are independently addressable by controller 110.For example, controller 110 can issue commands (or portions thereof) andany associated data to/from each of two or more data resources 112 to becarried out during the same or overlapping time periods. In accordancewith one embodiment, the controller 110 is configured to receive atleast one command and decompose the at least one command into aplurality of subcommands or “jobs.” A job can comprise a complete dataoperation for a command and/or can comprise a portion of an overall dataoperation for a command. For example, a single read or write command canbe decomposed into a plurality of jobs to be performed by the datamemory devices 112.

In one embodiment, the plurality of jobs are issued to a number of thedata memory devices 112 in a parallel manner. For instance, an exemplaryread command is decomposed into at least a first job and a second job.The first job pertains to a data access request for data stored in afirst data memory device 112 and the second job pertains to a dataaccess request for data stored in a second data memory device 112 thatis different than the first data memory device 112. The first and secondjobs are issued by the controller 110 to the first and second datamemory devices 112, respectively, and are performed by the devices 112during the same time period (or substantially the same time period) orduring overlapping time periods, in one example. The number of jobsgenerated by the controller 110 for a command can be a function of, forexample, the physical addresses of the data and the number of datamemory devices 112 to be accessed for the particular command.

Controller 110 is also configured to concurrently manage completioninformation for each of the issued jobs and any data associated with thecompleted jobs. For example, for a read command controller 110 isconfigured to track a number of data blocks required for completion ofeach job and to track a number of data blocks that have been obtainedfor each job. Further, for a particular job controller 110 can also beconfigured to track a completion status of at least one other job in theplurality of jobs. For example, as discussed above a command (and thecorresponding job decomposition) can have a defined or sequential orderthat is based on host requirements for the completion of the command(e.g., an order that the data is to be delivered to the host).Controller 110 is configured to concurrently manage the data memorydevices 112 such that data can be received from the data memory devices112 in the defined order and/or in an order that is different than thedefined order.

To illustrate, FIG. 3 is a diagram of an exemplary data structure 300that represents an exemplary command 302. While command 302 isillustrated as a single command, it is noted that command 302 cancomprise multiple commands for data operations, such as read and/orwrite operations.

Data structure 300 can be implemented in hardware, software and/orfirmware associated with controller 110. In one particular example, datastructure 300 comprises one or more linked-lists implemented in firmwareof controller 110.

The command 302 has been decomposed, for example by controller 110, intoa plurality of jobs (illustratively four jobs). Data structure 300includes a plurality of nodes 304-1, 304-2, 304-3, and 304-4(collectively referred to as “nodes 304” or “jobs 304”). In theillustrated embodiment, each of nodes 304 corresponds to a data accessrequest to a particular data memory device 112. For example, node 304-1represents a first job comprising a data access request for a firstportion of data stored in a first data memory device. Node 304-2represents a second job comprising a data access request for a secondportion of data stored in a second data memory device. Likewise, node304-3 and 304-4 represent third and fourth jobs for command 302. Asillustrated, command 302 requires a total of 14 blocks of data storedacross multiple data storage devices 112. Further, it is noted that eachof the jobs can pertain to data at different data storage devices 112and/or a plurality of the jobs can pertain to data at the same datastorage device 112.

Data structure 300 is illustratively a linked-list of job completioninformation, is utilized to manage completion of the jobs for command302, and is stored in controller 110, for example. In one example, thejobs for command 302 have an order that is defined based on the command302 and/or constraints on the subsequent transfer of data to the host.For example, in the illustrated embodiment the data for the first job(node 304-1) is to be released from the controller 110 to the hostbefore (or at least with) the data for the second job (node 304-2).Similarly, the data for the second job is to be release before (or atleast with) the data for the third job, etc.

The first node (e.g., node 304-1) in the data structure 300 is referredto as the “head” node. In accordance with one embodiment, the controller110 is configured to release data to the host when the job representedby the “head” node completes. On the other hand, when a job representedby a “non-head” node completes, the controller 110 does not release thedata to the host and instead stores the data (for example, in buffermemory 232) and indicates the stored data by modifying a preceding nodein the data structure 300.

Data structure 300 can thus be utilized by controller 110 to managecompletion of the jobs “out-of-order.” For example, using data structure300 controller 110 can receive and track data for the jobs in an orderthat is different than the order of the jobs illustrated in FIG. 3. Forinstance, controller 110 can receive and store the data for the secondjob (node 304-2) before the controller 110 receives any data for thefirst job (node 304-1). Subsequently, when the first job (node 304-1)completes (i.e., all the data for the first job is ready at thecontroller 110) the controller 110 can release the data for the firstand second job to the host at the same (or substantially the same) time.

In the illustrated embodiment, each node 304 in the linked-list pertainsto one of the jobs and can include fields such as, but not limited to,the following:

1) Requested // Number of total data blocks in the job 2) Complete // Arunning total of the number of data blocks completed (i.e., received atthe controller 110) for the job 3) ExtraRelease //Number of extra blocks(i.e., blocks in other jobs) to be released when the job is completed 4)JobStatus // Status of the job (i.e., InProgress, Done, Error, etc.) 5)Nextjob // Pointer to the next job in the list 6) JobInfo // Additionalinformation needed to complete the job

In the context of FIG. 3, node 304-1 includes a “requested” field 306that indicates the number of data blocks requested for the first job andincludes a “complete” field 308 that indicates the number of data blocksthat have been completed (i.e., received from the appropriate datastorage device 112) for the first job. Fields 306 and 308 can also berecorded in units of LBAs, pages, bytes, etc. When controller 110receives some or all of the data for the first job (node 304-1) from thedata storage device 112, controller 110 can store the data (for example,in buffer memory 232) and adjust the “complete” field 308. Once all ofthe data for the job has been received, the job status flag 312 ischanged to indicate that the job is done.

Node 304-1 also includes an “ExtraRelease” field 310 that representsdata for at least one other job (nodes 304-2, 304-3, and/or 304-4). Inaccordance with one embodiment, the “ExtraRelease” field 310 indicatesthe number of blocks of data acquired for completed jobs. In oneembodiment, the “ExtraRelease” field 310 for the first job (node 304-1)indicates the number of blocks of data acquired for completed jobs thatare immediately after the first job (node 304-1) in the defined joborder. In this manner, field 310 indicates the number of additional datablocks that are to be released with the data for the first job (node304-1) when the first job also completes.

Further, each node 304 can also include additional fields such as, butnot limited to, the number of sectors of data transferred to the host,the logical block addresses (LBAs) associated with the requested data,the physical block addresses associated with the requested data, etc.

FIGS. 4A and 4B illustrate a flow diagram 400 of an exemplary method forconcurrent data management using the exemplary data structure 300illustrated in FIG. 3. In the context of FIGS. 4A and 4B, each row ofnodes 304 illustrates a particular snapshot or instance in time. It isnoted that snapshots 420-432 have been chosen to highlight the datastructure 300 and are not intended to show every instance or datarelease opportunity.

At snapshot 420, the command has been decomposed into the plurality ofjobs represented by nodes 304; however, data has not yet been receivedfor any of the jobs. At snapshot 422, data for some of the jobs has beenreceived from the data memory devices. For instance, three blocks ofdata of the four requested blocks have been obtained for the first job(node 304-1). Four blocks of data of the four requested blocks have beenreceived for the second job (node 304-2). Two blocks of data of the fourrequested blocks have been received for the third job (node 304-3). Noblocks of data have been obtained for the fourth job (node 304-4). Node304-2 is modified to indicate that the second job has completed. Thecompleted non-head node 304-2 can be removed or collapsed into thepreceding node (head node 304-1). For instance, as can be seen atsnapshot 424, the node 304-2 has been removed and the “ExtraRelease”field of node 304-1 has been changed to indicate the four blocks of datathat were obtained for the completed second job (node 304-2).

At snapshot 426, all four data blocks for the third job (node 304-3)have been obtained. As illustrated in snapshot 428, node 304-3 iscollapsed into the preceding node (in this instance the head node304-1). The “ExtraRelease” field of node 304-1 is changed to indicatethat the job for node 304-3 has also been completed.

At snapshot 430, all four of the requested blocks of data for the firstjob (node 304-1) have been received. Because node 304-1 comprises thehead node, node 304-1 is removed and the data for the first job alongwith the data indicated in the “ExtraRelease” field (i.e., the data forthe second and third jobs) is released (e.g., transferred from the datastorage system to the host that issued the command) in an appropriateorder.

At snapshot 432, only the fourth job (node 304-4) remains; node 304-4comprises the head node. The data for the preceding jobs in the orderhave already been released to the host. At snapshot 432, the tworequested blocks of data have been obtained for the fourth job (node304-4). Because node 304-4 is now the head node, the data for the fourthjob is also released (i.e., transmitted to the host) along with any dataindicated in the “ExtraRelease” field. In this case, there is no extradata to release. Node 304-4 is removed.

In the embodiment of FIGS. 4A and 4B, a single read command isillustrated. In accordance with another embodiment, a plurality ofcommands can be concurrently managed. FIGS. 5A and 5B illustrate a flowdiagram 500 of an exemplary method for concurrent data management ofmultiple data commands.

Flow diagram 500 illustrates a first command 502 and a second command552. In the context of FIGS. 5A and 54B, boxes 522, 524, 526, and 528illustrate particular snapshots or instances in time. It is noted thatsnapshots 522, 524, 526, and 528 have been chosen to highlight the datastructure and are not intended to show every instance or data releaseopportunity.

As illustrated at snapshot 522, command A 502 has been decomposed into aplurality of jobs represented by nodes 504-1, 504-2, and 504-3. CommandB 552 has been decomposed into a plurality of jobs represented by nodes554-1 and 554-2. It is noted that each of the jobs represented by nodes504-1, 504-2, 504-3, 554-1, and 554-2 can comprises data access requestsfor different data memory devices. Alternatively, or in addition, two ormore of the jobs can comprise data access requests to the same datamemory device. For instance, the jobs represented by nodes 504-2 and554-1 can be issued to the same data memory device.

At snapshot 524, the job represented by node 504-2 has completed and thenode 504-2 is collapsed into the preceding node, in this case head node504-1. Similarly, the job represented by node 554-2 has completed andthe node 554-2 is collapsed into the preceding node, in this case headnode 554-1.

At snapshot 526, the jobs represented by nodes 504-3 and 554-1 havecompleted. Because node 554-1 is the head node, the data blocks for thejob represented by node 554-1 are released, along with the dataindicated in the “ExtraRelease” field (in this case, the two blocks ofdata from node 554-2). At this point, command B 552 has completed andall the data for command B 552 has been transferred to the host. Becausethe data blocks for node 554-2 were already received and stored by thecontroller (and represented by the “ExtraRelease” field in node 554-1),the data storage system can transfer all of the data for the commandwithout having to wait for the data to be acquired from the data storagedevices for jobs 554-1 and 554-2 “in-order”.

At snapshot 528, the job represented by node 504-1 has completed.Because node 504-1 is the head node, the data for job 504-1 is releasedalong with the data indicated in the ExtraRelease field (in this casethe data for nodes 504-2 and 504-3). Likewise, the controller 110 is notrequired to wait for the data for the subsequent nodes 504-2 and 504-3to be transferred from the data memory devices as this data has alreadybeen transferred to the controller and is represented in theExtraRelease field of the head node.

In one embodiment, although command A 502 arrived at the controllerbefore command B 552, the data handling required by command A 502 causescommand A 502 to take longer to complete than command B 552. Thus,although commands are received (e.g., enqueued) in a particular order,their order to “attach” to the host bus for data transfer can bedifferent. This can allow for a more efficient transmittal of data as anearlier received command that takes a significant time to complete doesnot monopolize the host bus.

In accordance with one embodiment, one or more write commands arereceived by the controller and a plurality of jobs are generated for thewrite command(s). For the jobs generated for write command(s), theExtraRelease field generally relates to the amount of buffer space thatcan be reallocated for other uses after the data for the job has beenwritten to the appropriate data memory devices and the job hascompleted.

In some systems, the host bus protocol may not allow multipleconnections for data transfers for a particular command. Thus, when adata transfer for a first command begins, all of the data for the firstcommand must be transferred before a data transfer for a second commandbegins. In accordance with one embodiment, in the constrained case(where the host bus protocol does not allow multiple connections for acommand) the controller is configured to connect the command to the busfor data transfer that is most likely to maximize the number releases ofdata per unit time until the end of the command. In one example, this isimplemented by assuming that all commands are similarly sized, andattaching the command for data transfer whose data operation was startedfirst on the back-end data storage resources. The system then givespreferential treatment to that command to limit or prevent the commandfrom stalling the host bus, for example while the back-end data storageresources are accessed for the command. In another embodiment, aparticular read command is not selected for attachment to the host busfor data transfer if the particular read command still requires morethan one data transaction per back-end data storage resource to completethe read command.

In one embodiment, in the constrained case (where the host bus protocoldoes not allow multiple connections for a command, for examplezero-buffer offset mode of first-party DMA in SATA) the queue entry foreach command includes a pointer to the data structure (for example, datastructure 300, 500) representing completion information for the command.The system hardware can utilize the pointer to keep track of thecompletion information for the command that is currently attached to thehost bus and use the completion information to throttle transmission ofdata packets to the host. In one example, the system hardware uses thecommand completion information to throttle automatic transmission ofdata packet that must be equal to a maximum packet payload (e.g., 8Kibibytes (KiB)) or the remainder of the host command, whichever issmaller.

In one embodiment, system firmware can pre-emptively make the selectionof the command to connect to the host bus using the completioninformation (for example, if the firmware determines that a particularread command is not going to need more than a single data operation perback-end resource and that none of the resources are busy servicingother requests). In one example, the system firmware selects a commandto connect to the host bus when the completion information for thecommand indicates that the head node of the linked-list associated withthe command has completed.

In one embodiment, in the unconstrained case (where the host busprotocol allows multiple connections for a command), a command can beconnected to the host bus for data transfer whenever a head node for acommand completes and the “Complete”+“ExtraRelease” fields for the headnode indicate an amount of data exceeding a threshold established by thefirmware, or is equal to the remainder of the requested data blocks forthe command, whichever is smaller.

In accordance with one embodiment, the data storage system is configuredto prioritize read commands ahead of write commands. In this manner, theread commands may be easier to predict with respect to their readinessto attach to the host bus. The write operations can be utilized to fillin the host bus bandwidth between the read command data transfers, asavailable.

In one embodiment, the data storage system is configured to abort awrite operation that is in progress if the abort is issued before thewrite operation has reached as particular point. For instance, to reduceread latency a write operation can be aborted in favor of a data readoperation as long as the actual “program” flash command for the writeoperation has not been issued. However, this may cause some wastedprocessing or bus-transferring capacity on the back-end. Thus, in oneembodiment a threshold can be tuned to adjust how far into the writeoperation an abort may be affected. In one example, the threshold isadaptively tuned based on the amount of write operations and/or readoperations in the command queue. For instance, if there are a largenumber of write operations in the queue, but very few read operations,the controller can be configured to increase the read latency tomore-preferentially process the write operations. This can beadvantageous in situations where the host is waiting for writeoperation(s) to complete.

FIG. 6 is a diagram illustrating another embodiment of a data structure600 representing command completion information for an exemplary command602. While command 602 is illustrated as a single command, it is notedthat command 602 can comprise multiple commands for data operations,such as read and/or write operations.

Data structure 600 can be implemented in hardware, software, and/orfirmware associated with controller 110, for example. In one particularembodiment, data structure 600 is implemented in hardware of controller110 and includes a fixed number of entries for a predefined or maximumnumber of allowed jobs. The fixed number of entries of structure 600relates to the number of jobs that can be outstanding and concurrentlymanaged by controller 110 at a time. The number of entries of structure600 can be based on constraints of the data storage system, such as thephysical architecture of the controller, channels, and/or storagedevices, for example.

To illustrate, in the embodiment of FIG. 6 data structure 600 includes acommand descriptor 604 and a fixed number of available job descriptors606 (illustratively four job descriptors 606). The command descriptor604 is indicative of the overall command 602 while each job descriptor606 is indicative of a particular job for the command 602. The number ofavailable job descriptors 606 can be based on a number of hardwareresources for carrying out data command 602 (e.g., a number of datamemory devices 112, a number of channels 114, etc.). The number of jobsthat can be concurrently managed by data structure 600 is, in oneembodiment, equal to the number of available job descriptors 606.Additional jobs for the command (for example, when the number of totaljobs for command 602 exceeds the number of available job descriptors606) can re-use job descriptors 606 from completed jobs. This can beadvantageous when the command 602 requires more jobs than can be trackedat one time with the available job descriptors 606.

While four job descriptors 606 are illustrated, it is noted that inother embodiments data structure 600 can include more than or less thanfour job descriptors 606. For example, data structure 600 can includetens, hundreds and/or thousands of job descriptors 606. Each one of jobdescriptors 606 is associated with (e.g., mapped to) a particular one ofdata memory device (e.g., logical unit number (LUN)). For instance, jobdescriptor 606-0 is associated with LUN 0, job descriptor 606-1 isassociated with LUN 1, job descriptor 606-2 is associated with LUN 2,and so forth.

Upon reception of data command 602 that requires transfer to or from thedata memory devices (e.g., data memory devices 112), the controller 110generates the command descriptor 604. In the illustrated embodiment, thecommand descriptor 604 includes fields such as, but not limited to, thefollowing:

1) StartAddress // Address (e.g., LBA) associated with the start of thecommand 2) TotalBlocks // Number of blocks (e.g., sectors, 8-byte units,bytes, etc.) in the command 3) BlocksRemaining // Number of blocksremaining to be requested as jobs 4) BlocksAvailable // Number of blocksavailable from the head of the command (e.g., the head job descriptornode) 5) JobReadPointer // Pointer to oldest uncompleted job descriptor6) NextJobPointer // Pointer to job descriptor to be filled in next

The StartAddress and TotalBlocks fields are initialized based on thereceived command 602. The BlocksRemaining field is initialized to thesame value as the TotalBlocks. The BlocksAvailable, JobReadPointer, andNextJobPointer fields are all initialized to zero. In the exampleillustrated in FIG. 6, command 602 includes a start address of LBA 0 andhas a total number of requested blocks equal to 14.

Next, hardware and/or firmware of the controller 110, for example,perform mapping lookups sequentially through the transfer address rangeof command 602 to calculate the size in blocks of each required job. Forinstance, the controller 110 determines the number of jobs needed (andthe number of required blocks for each job) to complete the command 602based on the manner in which the data storage devices 112 need to beaccessed for the command 602. In one example, the controller 110decomposes the command 602 into a plurality of jobs based on theaddresses of the data in the plurality of data storage devices 112. Eachjob descriptor 606 pertains to a particular logical unit number (LUN)for the job associated with that job descriptor 606.

As each job size is determined, the job descriptor 606 is initialized.The “Required” field of the job descriptor 606 is set to the size of thejob (for example, the number of requested data blocks for the job) andthe “Complete” field for the job descriptor 606 is initialized to 0. The“Complete” field indicates the amount of data (e.g., number of datablocks) that has been acquired by the data memory device for the joband/or has been transmitted from the data memory device to thecontroller 110. Data received by the controller 110 can be stored in abuffer or cache, for example.

FIGS. 7A and 7B illustrate a process of initializing the job descriptors606 and utilizing the data structure 600 to concurrently manage the jobsissued to the data resources.

As illustrated at snapshot 620, controller 110 issues the first job (job0) for the command 602 and initializes the job descriptor 606-0 for thefirst job (job 0). Because job 0 requires four blocks of data, theRequired field of job descriptor 606-0 is set to 4. The Completed fieldof job descriptor 606-0 is initialized to 0 and the BlocksRemainingfield of the command descriptor 604 is decreased by the size of theinitialized job 0. The NextJobPointer field is incremented to indicatethat job descriptor 606-1 is the next job descriptor to be filled in(i.e., job 1 is the next job to be issued). In one embodiment, when theNextJobPointer field of command descriptor 604 reaches the end of theavailable job descriptors 606, the NextJobPointer is wrapped to 0 topoint to job descriptor 606-0. In this instance, all available jobdescriptors 606 have been initialized and the controller is configuredto reuse job descriptors 606 for any additional jobs required for thecommand 602. The loading of the job descriptors 606 continues until theBlocksRemaining field of the command descriptor 604 reaches 0 or theNextJobPointer equals the JobReadPointer (all job descriptors 606 forthe command are in use).

Once a job descriptor 606 is initialized, a media transfer associatedwith that job descriptor can be initiated. For example, at snapshot 620the data transfer associated with job 0 can initiate even though otherjobs for the command have not been issued.

At snapshot 622, the job descriptors 606-1, 606-2, and 606-3, associatedwith jobs 1, 2, and 3, respectively, have been initialized. Thus, all ofthe jobs have been issued and all of the command descriptors 606 havebeen initialized. However, no data blocks have been acquired for thejobs. The NextJobPointer of command descriptor 604 has been incrementedto the next job pointer, in this case job 4. However, in the case whereonly four job descriptors are available, the NextJobPointer can beconfigured to wrap to the first job descriptor 606-0.

At snapshot 624, some of the data for the command has been acquired fromthe data memory devices. The data memory device associated with job 0has acquired 3 blocks of data, the data memory device associated withjob 1 has acquired 4 blocks of data, and the data memory deviceassociated with job 2 has acquired 2 blocks of data. As the data isacquired, the data is transferred to the controller for storage, forexample in a buffer, while additional data is obtained from the datamemory devices. The job descriptors 606 are updated accordingly toreflect the number of acquired data blocks for the jobs. As illustratedat snapshot 624, the BlocksAvailable field of command descriptor 604 isupdated to indicated that the 3 blocks of data have been acquired forthe head job descriptor (in this case job 0). These 3 blocks of data canbe transferred to the host at snapshot 624 or can be held until moredata is acquired.

At snapshot 626, the jobs associated with job descriptors 606-1 (job 1)and 606-2 (job 2) have completed. In one embodiment, the data for jobs 1and 2 is provided to the controller before the completion of job 0.

At snapshot 628, job 0 associated with the job descriptor 606-0 (whichin this case is the head job descriptor) has completed as all 4 requireddata blocks have been acquired from the associated data memory deviceand provided to the controller 110. At this point, the BlocksAvailablefield of the command descriptor 604 is updated to reflect that 13 blocksof data (4 blocks from job 0, 4 blocks from job 1, 4 blocks from job 2,and 1 block from job 3) have been acquired by the controller 110 and areavailable for transfer to the host, for example. In one embodiment,these 13 available blocks of data are “released” or transferred from thecontroller 110. Alternatively, the controller 110 can be configured todelay the data transfer to the host until the remaining data for thecommand has been acquired. At snapshot 628, the JobReadPointer field hasalso been updated to reflect that the next or oldest uncompleted jobdescriptor is now job descriptor 606-3.

At snapshot 630, the last block(s) of data for job descriptor 606-3 areacquired by controller 110 and the data is “released” from thecontroller 110. The JobReadPointer field is updated to reflect that jobdescriptor 606-3 has completed and to indicate the next uncompleted jobdescriptor (in this case the JobReadPointer wraps to 0).

In accordance with one embodiment, each job descriptor 606 is mapped toand associated with a particular data memory device having a particularlogical unit number (LUN), for example. If the command(s) 602 do notcomprise a job for one or more of the data memory devices, thecorresponding job descriptors 606 for those data memory devices includea “next job” pointer. The “next job” pointer operates as a placeholderand instructs the controller to advance or skip that job descriptor.Alternatively, or in addition, the Required and Completed fields for thejob descriptors that are not being used for the command(s) can be set to“0”'s to indicate that the job descriptors are not being used and shouldbe skipped.

In accordance with one embodiment, the described data structures (e.g.,data structures 300 and 600) enable tracking and management of jobsacross a plurality of concurrent resources. In one aspect, one or morereceived commands are decomposed into a plurality of jobs, each jobbeing indicative of a portion of the command(s). The jobs have an orderthat is defined based on, for example, the addresses of the data in thedata storage devices. In one example, the order of the jobs is afunction of a host-expected or host-acceptable order of the data. Forinstance, host protocol can require that the data for the jobs of thecommand are received by the host in the defined order. The datastructure(s) are utilized by the controller 110 of the data storagesystem to concurrently manage the jobs such that the controller 110 canreceive the data for the jobs in an order that different (e.g.,out-of-order) than the defined order for the jobs. This can beadvantageous as the controller 110 can acquire the data from the datamemory devices as the data is made available at the data memory devices;the controller does not have to wait for the jobs to complete at thedata memory devices in the defined order.

The implementations described above and other implementations are withinthe scope of the following claims.

1. A method in a data storage system, the method comprising: defining aplurality of jobs for a command received from a host, each of theplurality of jobs being associated with one or more of a plurality ofdata storage resources of the data storage system, the plurality of jobshaving a defined order that is a function of addresses of data in theplurality of data storage resources; issuing the plurality of jobs tothe associated data storage resources; receiving information from thedata storage resources for the plurality of jobs, the information beingreceived by a controller of the data storage system for the plurality ofjobs in an order that is different than the defined order; andtransmitting the received information to the host for the plurality ofjobs in the defined order.
 2. The method of claim 1, wherein theplurality of data storage resources comprise a plurality of solid-statedata memory devices, each of the plurality of solid-state data memorydevices being identified by a logical unit number (LUN).
 3. The methodof claim 1, comprising receiving the command for a data read operationof a plurality of data blocks from the data storage resources, whereinreceiving information comprises receiving data from the data storageresources for at least some of the plurality of jobs, the defined orderfor the plurality of jobs comprising an order defined based on thelogical block addresses of the plurality of data blocks.
 4. The methodof claim 3, wherein transmitting the received information comprisestransmitting the data blocks for the jobs to the host in the definedorder, the defined order comprising a host expected order of the data.5. The method of claim 1, comprising receiving the command for a datawrite operation of a plurality of data blocks to the data storageresources, wherein receiving information from the data storage resourcescomprises receiving an indication of an amount of buffer space to bereallocated.
 6. The method of claim 1, and further comprising utilizinga data structure to concurrently manage each of the plurality of jobs.7. The method of claim 6, wherein the data structure comprises alinked-list of completion information for the plurality of jobs, thelinked-list comprising a plurality of nodes, each node of the pluralityof nodes pertaining to one of the plurality of jobs.
 8. The method ofclaim 7, wherein each job is represented by data in a node of thelinked-list, the data for each job comprising: a first field indicativeof data blocks required for the job; a second field indicative of datablocks acquired for the job; and a third field indicative of completionof at least one other job in the defined order.
 9. The method of claim6, wherein the data structure comprises a command descriptor and aplurality of job descriptors, each job descriptor pertaining to one ofthe jobs and including information indicative of the number of datablocks for the job, wherein the command descriptor includes datacomprising: a first field indicative of a number of data blocks for thecommand; a second field indicative of a number of data blocks acquiredfrom the data storage resources for the plurality of jobs; and a thirdfield comprising a job read pointer indicative of a first job in thedefined order that has not been completed.
 10. The method of claim 1,wherein issuing the plurality of jobs comprises issuing a first job inthe defined order to a first data storage resource and issuing a secondjob in the defined order to a second data storage resource, and whereinreceiving information comprises receiving information for the second jobfrom the second data storage resource before the first job is completedat the first data storage resource, and wherein when the first job issubsequently completed at the first data storage resource, transmittingthe information for both of the first job and the second job to thehost.
 11. A data storage system comprising: an interface for receiving acommand for a data operation; a plurality of data memory devices; and acontroller configured to generate a plurality of jobs for the command,issue the plurality of jobs to the data memory devices, and receiveinformation from the data memory devices for the plurality of jobs,wherein each of the plurality of jobs are associated with one or more ofthe data memory devices, the plurality of jobs having a defined orderthat is a function of addresses of data in the data memory devices, andwherein the information is received at the controller from the datamemory devices for the jobs in an order that is different than thedefined order.
 12. The data storage system of claim 11, wherein theinterface comprises a host interface for communication with a hostsystem, and wherein the controller is configured to transmit theinformation for the plurality of jobs to the host, using the interface,in the defined order.
 13. The data storage system of claim 11, whereinthe controller is configured to utilize a data structure to concurrentlymanage each of the plurality of jobs for the command.
 14. The datastorage system of claim 11, wherein the plurality of data memory devicescomprise a plurality of solid-state data memory devices, each of theplurality of solid-state data memory devices being identified by alogical unit number (LUN).
 15. A data storage device comprising: aplurality of data storage resources; and a controller configured toidentify at least one command for a data operation and generate aplurality of jobs for the data operation based on the at least onecommand and the plurality of data storage resources, wherein thecontroller is configured to issue the plurality of jobs to the pluralityof data storage components and includes a data management componentconfigured to concurrently manage completion information pertaining tocompletion of the plurality of jobs.
 16. The data storage device ofclaim 15, wherein the data management component is configured togenerate a data structure including completion information indicative ofcompletion of the plurality of jobs.
 17. The data storage device ofclaim 16, wherein the data structure includes a linked-list includingdata for each job, the data for each job comprising: a first fieldindicative of data blocks required for the job; a second fieldindicative of data blocks acquired for the job; and a third fieldindicative of completion of at least one other job of the plurality ofjobs.
 18. The data storage device of claim 17, wherein the plurality ofjobs have a defined order that is a function of the at least onecommand, and wherein for each job the first field is indicative of datablocks needed to complete the job, the second field is indicative of anumber of data blocks completed for the job, and the third field isindicative of a number of data blocks completed for subsequent jobs inthe defined order.
 19. The data storage device of claim 16, wherein thedata structure comprises a command descriptor and a plurality of jobdescriptors, each job descriptor pertaining to one of the jobs andincluding information indicative of the number of data blocks for thejob, wherein the command descriptor includes data comprising: a firstfield indicative of a number of data blocks for the command; a secondfield indicative of a number of data blocks acquired from the datastorage resources for the plurality of jobs; and a third fieldcomprising a job read pointer indicative of a first job in the definedorder that has not been completed.
 20. The data storage device of claim15, wherein the plurality of data storage resources comprise a pluralityof solid-state data memory devices, each of the plurality of solid-statedata memory devices being identified by a logical unit number (LUN).